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Abstract 

The American Society for Training and Development (ASTD) defines human performance technology 
(HPT) as “a systematic approach to analyzing, improving, and managing performance in the workplace 
through the use of appropriate and varied interventions. ” The first step in this approach is the performance 
analysis ( Gilbert, 1 978). In this step, the performance technologist/consultant works collaboratively with the 
client to examine the current situation. Performance gaps or deficiencies are identified and are prioritized 
according to the needs of the client (ISPI, 2004). This case study describes a project in a graduate level HPT 
class at a Midwestern university. A team of graduate students was formed to address computer security issues 
for the client organization. The project team implemented a performance analysis process (Define, Analyze, and 
Select) as described by Schaffer and Douglas (2004). This process incorporated tools and frameworks such as 
the Performance Relationship Map (Robinson and Robinson, 1995) and the Performance Pyramid (Wedman 
and Graham, 1998, 2004). The significance of the project is that it provided a real world context in which the 
project team and the client could learn about HPT processes. The experiences were enriched by the request 
from the client to continue the HPT process after the analysis project. 

Introduction 

“Think performance, not training!” (Robinson & Robinson, i995, p. 6). Human Performance 
Technoiogy (HPT) is a fieid of professionai practice which is project-based and focused on workpiace 
effectiveness (ISPI, 2004; Stoiovitch & Keeps, 1999). The application of procedures is derived from scientific 
research and professionai experience and is appiied to the soiution of practicai probiems. The American Society 
for Training and Deveiopment (ASTD) defines performance technoiogy as “a systematic approach to anaiyzing, 
improving, and managing performance in the workpiace through the use of appropriate and varied 
interventions.” 

Many names have been given to this fieid, inciuding human performance technoiogy (Stoiovitch & 
Keeps, i999), human performance improvement (HPI) (Rothweii, Hohne and King, 2000), human performance 
enhancement (Rothweii, 1996) performance consuiting (Robinson & Robinson, 1995), performance engineering 
(Dean i994; Giibert, i978), performance technoiogy (Hariess, i992), and so forth. In addition to the various 
names, many different performance improvement process modeis exist. 

The first step in the HPT approach is the performance anaiysis, in which the performance technoiogist 
works coiiaborativeiy with the ciient to examine the current situation at one or more of the foiiowing ieveis: 
societai, organizationai, process, work group, or individuai. Performance gaps or deficiencies are identified and 
are prioritized according to the needs of the ciient (ISPI, 2004). 

Schaffer and Douglas (2004) developed a framework for object-oriented performance anaiysis for the 
Automated Object-Oriented Performance Anaiysis Project (AOOPA). This framework recommends that 
organizations not bypass the probiem-soiving process by negiecting the definition and anaiysis of a 
probiem/opportunity or by skipping directiy to the seiection of a singie soiution. According to this framework, 
the most basic eiements in performance anaiysis are three iterative phases: define, anaiyze, and seiect. 

The major tasks for each phase are as foiiows: 

■ Define: Define the opportunity or probiem 

0 Start with ciear statement of the opportunity or probiem 
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0 Identify gaps between what is and what should be at the organizational level 
0 Communicate with stakeholders using visual models 
0 Provide a rationale for decisions 

■ Analyze: Analyze gaps between what should be and what is at performer level 

0 Develop a performance model as foundation for development of data collection instruments to 
identify current performance and work environment barriers 
0 Review data with clients and collaborate to identify gaps 
0 Provide a rationale for decisions 

■ Select: Select solutions and recommend actions 

0 Map possible causes for gaps to possible solutions 
0 Collaborate with clients to prioritize 
0 Blend solutions most appropriate to organizational context 
o Report recommendations 
o Provide a rational for decisions 

This paper presents the experiences of a team of graduate students in Educational Technology at a 
university in the Midwest while conducting a performance analysis in a HPT course project. The team utilized 
the consolidated performance analysis with three major phases - define, analyze, and select; and integrated 
other organizational frameworks and models, including Robinson & Robinson’s (1995) Performance 
Relationship Map and Wedman and Graham’s (2004) Performance Pyramid, into the process to complete the 
performance analysis of computer users who lived in residence halls. The process and results of the 
performance analysis, as well as challenges, limitations and lessons that the performance analysis team learned 
are discussed in this paper. The team’s completion of the project provided them with the experience necessary 
to answer the following questions, which they discovered are crucial to the performance analysis process: 

• What can be done to help the client understand the performance analysis process? What actions 
can be taken to ensure that the client is involved in performance analysis? 

• When analyzing the causes of the performance gaps, what can be done to encourage clients to 
investigate causes other than the lack of skills and knowledge (or solutions other than training)? 

• What is the next step after the performance analysis? 

The Case 

Overview 

The study was a class project in a graduate level HPT class. A project team of four graduate students 
with two doctoral and two master’s students was formed. All members of the team were novices in the HPT 
field. The major concepts, including the definitions to the field, systems theory, HPT frameworks and models, 
and performance analysis were introduced before the beginning of the project. The client organization was one 
of the groups in the security and policy department, known as Security Outreach and Training, under the 
computer services at a midwestern university. The client was responsible for proactively and reactively 
combating the information security problems of the university population. 

The performance analysis process began in October 2003 with an introductory meeting with the 
manager of the client organization. The project team completed the performance analysis in December 2003. 

Define Phase 

An initial client meeting with the client team was called for project alignment. The project team 
immediately began working through the Performance Relationship Map (Figure 1) with the client using an 
interview guide that addressed key performance analysis factors. The business problem was identified as 
“Insecure residence hall computer network computers are generating an unnecessary workload for the security 
and policy department.’’ The business need was defined as “The university community will have a secure 
computing infrastructure. The target performers were identified as the computer network users who lived in the 
residence halls. The Operational Results and the On-The-Job Performance Should’s on the Performance 
Relationship Map were documented after the desired organizational goals and desired individual performance 
goals were discussed. 
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Figure 1. The Performance Relationship Map adapted from Robinson & Robinson (1995) 
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In addition, a system model (a visual representation of the organizational process with inputs and 
outputs) was sketched together with the client in order to identify all the key stakeholders and their relationships 
surrounding the business problem. Resources and available support from the client were also discussed and 
arranged, such as the available data for the current operational results. Finally, the data collection process was 
brainstormed with the client. 

After the meeting, a Statement of Work, which summarized the business problem and need, key 
stakeholders, resources for the performance analysis, target performers, and initial performance gaps based on 
available data and discussions in the initial meeting, was prepared and emailed to the client for confirmation. 

Analyze Phase 

Data Collection The analyze phase started with the analysis of avaOable data provided by the client. 
The client had previously collected data about the number of machines compromised in particular outbreaks and 
vulnerabilities, the percentage of email messages received by the target performers that were classified as spam, 
and the number of machines in the mail server infected by a recent virus. The project team used this data to 
begin completing the Operational Results -Is box on the Performance Relationship Map. 

Next, due to the population size of 10,500 performers, requirements for quantitative data, and the need 
for confidentiality of respondents, the project team selected a questionnaire as the instrument for additional data 
collection. In addition, a performance model (Robinson & Robinson, 1995), which detailed the competencies 
and computer security best practices for the performers, was drafted. As mentioned by Schaffer (2000), Dean 
and Ripley (1997) indicated that the ability to integrate and synthesize useful frameworks, processes and data 
that link the major systems and subsystems within and outside the boundaries of an organization is a critical 
skill set of performance improvement specialist. The development of the questionnaire is an example of this, as 
it required the integration of different process models and organizational frameworks. It was designed and 
revised based on the discussions during the initial client meeting, the initial performance relationship map, 
existing documents developed by the client, the performance model, as well as the comments and suggestions 
from the client and the project advisor. 

In an effort to simplify the data analysis process, the project team developed the questionnaire so that it 
provided insight into both the existence and also the causes of existing performance gaps. To achieve this, the 
questions were crafted so that they assessed the performers’ use of the best practices and also emphasized 
several of the building blocks, such as knowledge and skills, tools and equipment, etc. in the Performance 
Pyramid (Figure 2). The questions that addressed the Performance Pyramid blocks provided the basis for 
determining the environmental barriers and the internal causes on the Performance Relationship Map. 
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Figure 2. The Performance Pyramid (Wedman & Graham, 2004) 




After development, the project team sent the questionnaire to the client for review. The client’s 
feedback was incorporated. The questionnaire was then finalized and developed into a web-based questionnaire. 
Because the target performers were computer network users who lived in the residence hall, and because the 
majority of the network users living in the residence halls were undergraduate students, the subjects were 
chosen to be the undergraduate students from a 100-level computer technology class, a 200-level education 
class, and a 100-level engineering class. The survey was conducted for one week and 173 complete responses 
were collected. 



Data Reporting The data collected were processed to identify current individual performance. A 
summary of data, pie charts and bar charts were prepared for the data review meeting as visual representations 
that were used to compare quantities, amounts, and proportions. A Pareto Chart, which is a specialized type of 
bar chart that organizes the data from highest to lowest (or lowest to highest), was also used. This chart helps to 
determine the causes with the most impact; and “is a highly useful way to establish priorities on problems or 
causes by surfacing and displaying those which are most problematic” (Rothwell, Hohne and King, 2000, p. 

79). Instead of diagramming the responses for causes in Pareto Charts, the gaps were diagramed in order to help 
identify the unacceptable gaps. The causes were then arranged according to the building blocks of the 
Performance Pyramid for the ease of distinguishing the type of cause, such as knowledge & skills, rewards & 
incentives, etc. (Figure 3). 
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Figure 3. Responses to Question 3: “Please tell us why you don’t update antivirus software at least once per 
week” grouped by Performance Pyramid block 
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• Q3_l_3: I 
don’t have the 
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(6.4%) 

• Q3_l_6: 1 
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how to get 
help. (3.5%) 

• I don’t have it. 

( 0 . 6 %) 



Data Review Meeting During the data review meeting, the preliminary data report with a summary of 
data, descriptive statistics, i.e. frequencies, drafts for the performance model, Pareto charts, the performance 
relationship map, and results related to causes organized according to the Performance Pyramid was presented 
to the clients. The data from the questionnaire, as well as the data that was previously collected by the client, 
were used to complete the On-the-Job performance IS data on the Performance Relationship Map (Figure 4). 

Through discussion, the client and the project team reached a group consensus on unacceptable gaps 
and environmental barriers. The project team and the client brainstormed some potential causes for the gaps in 
addition to those supported by the data. Frequently, the project team had to redirect the client so that causes 
other than a lack of knowledge or skills were considered. The project team did this by referencing the 
Performance Pyramid and asking the client about potential causes from specific blocks other than knowledge 
and skills. 

In addition, the project team also had to redirect the client to focus on the causes instead of the 
solutions during the meeting, as the client representatives desired to jump straight to the solutions once they saw 
the data that displayed the performance gaps. 

After the meeting, the project team prepared a cause prioritization worksheet, which listed the potential 
causes for the performance gaps, and provided to spaces for the two representatives of the client organization to 
rank those which they felt were important to address. Originally, this worksheet was to be discussed during the 
data review meeting. However, time did not allow for this discussion. Instead the worksheet was emailed to 
the client. Besides requesting that the client provide a rationale for the prioritization of the causes, the team also 
asked the client to list the potential solutions if applicable. Unfortunately, the representatives of the client 
organization did not provide much of a rationale or possible solutions. The time constraint that forced the 
project team to conduct this part of the performance analysis by email negatively affected the response of the 
client. 
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Figure 4. Completed Performance Relationship Map 



Organizational 


Individual (ResNet Users in Residenee Halls) 


Desired Results (Should) 

♦ 5% or less of all machines in ResNet 
compromised in any outbreak of 
vulnerability. 

♦ The mailhub virus statistic goes down to 
zero. 

^ 20% of all emails are spam. 


Desired Results (Should) 

♦ 100% of users have strong passwords. 

♦ 100% of users change security settings to the highest 
level that works for the website that they want to 
browse. 

♦ 100% of users disable or set cookies to be discarded 
when a website is closed. 

♦ 100% of users choose not to download unknown files 
or programs when browsing the Internet. 

♦ 100% usage of firewall. 

♦ 100% of users update anti-virus software at least one 
per week. 

♦ 100% of users apply software and/or system updates 
and/or patches regularly. 

^ 100% usage of anti-virus software. 

♦ 100% of users do not give out email addresses or 
personal information to suspicious websites. 


GAP 


GAP 


Organizational 


Individual (ResNet Users in Residenee Halls) 


Current Results (Is) 

♦ Approximately 10-20% of all machines (or 
1000-1500 machines of 10, 500 registered 
hosts) on ResNet compromised in the 
outbreak of RPC DCOM worms. 

♦ Approximately 1000 hosts were infected at 
peak infection. 

♦ Approximately 40% of all emails are spam 
(i.e. 400,000 of 1 million of messages 
received per day in the Purdue mailhub). 


Current Results (Is) 

(Based on data collected from the survey) 

♦ 8.1% of users change password regularly (Q6_6). 

♦ 36.4% of users change security settings to the highest 
level that works for the website that they want to 
browse (Q7). 

♦ Strong password: 

^ 41 % of users disable guest login 

(Q6_10). 

^ 42.2% of users use different 

passwords for different accounts 
(Q6_7). 

^ 45.7% of users don’t set computer 

to remember passwords (Q6_8). 

^ 52% of users use keys next to each 

other on the keyboard (Q6_4). 

^ 57.8% of users don’t set computer 

to automatic login (Q6_9). 

^ 65.9% users do not use personal 

information like SSN, biithday, 
names and etc. (Q6_3). 

^ 68.2% of users use 8 or more 

characters (Q6_l). 

y 76.9% of users don’t share password 
with anyone (Q6_5.) 

^ 83.2% of users use letters, numbers 

and other characters (Q6_2). 

♦ 42.2% of users disable or set cookies to be discarded 
when a website is closed (Q8). 

♦ 50.9% of usage of firewall (Q4). 

♦ 70.5% of users update anti-virus software at least one 
per week (Q3). 

♦ 85% of users apply software and/or system updates 
and/or patches regularly (Q5). 

♦ 86.1% of users choose not to download unknown 
files or programs when browsing the Internet (Q9). 

♦ 93.1% of usage of anti- virus software (Q2). 

♦ 94.8% of users do not give out email addresses or 
personal information to suspicious websites (OlO). 
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Select Phase 

Select Solution After receiving the Cause Prioritization Worksheet from the client, the project group 
discussed solution types that would be the most appropriate and developed solutions accordingly. Solutions 
were brainstormed with a job aid for matching causes with possible solutions based on cause type. For example, 
when dealing with lack of knowledge and skills, training, job aids, and feedback systems were listed as the 
appropriate solution categories. A list of proposed solutions to the client can be found in Figure 5. 

Figure 5. List of proposed solutions 



List of Proposed Solutions 

Cause type: Expectations and Feedback, Desire and Self-Concept 
Solution type: Feedback Systems 

■ Implement a software module to ensure the use of a strong password 

■ Send a personalized email alert during major vims outbreak to remind performers to update their 
anti- vims software 

■ Send a pamphlet emphasizing free security products available* 

■ Send a personalized email reminder to change passwords once or twice during the semester 

■ Email a monthly news release that describes how computers were compromised and the resulting 
problems due to weak or unchanged passwords or lack of or improper firewall use 

Cause type: Knowledge and Skills 
Solution Type: Training 

■ Deliver a tutorial when the performers sign for their ResNet accounts to teach about anti-virus software 
use and firewall use* 

■ Present ITaP security resources and security best practices in dorm orientations* 

Solution Type: Job Aids 

■ Include a checklist with the characteristics of a strong password on the “Change Password” screen 

■ Include a security best practices checklist in the Purdue Mortar Board (student daily planner) 

■ Include a “how-to” job aid in the personalized email reminder to change passwords (mentioned 
above) 

Cause type: Rewards and Incentives 

Solution Type: Reward and Recognition Systems 

■ Give away free gifts (e.g. mouse pads, can-holders “koozies”) with security best practices 
checklists printed on them* 

■ Allow students who proceed through the tutorial (mentioned above) to enter a drawing for free 
computer hardware or software* 

*The performance analysis team suggested to the client that these solutions be included in the blended solution of 
holding “Security Awareness Weeks” at the beginning and/or end of each semester 



The potential solutions were then rated by three of the four team members based on the following four 
solution selection criteria: opportunity (an organization-level support and commitment), capability (the 
collective knowledge and skills of an individual, department or organization), collaboration (the level of user 
involvement in adoption, adaptation, and implementation processes), and motivation (the perception or attitude 
potential adopters and stakeholders have of the attributes of the solution). Categories were rated using a 5-point 
scale (1 for Disagree and 5 for Agree). In addition, the estimated cost for each potential solution was also 
estimated using a 5-point scale (1 for the most cost-effective). The averages for the ratings from the three 
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project members were calculated. The rationales for ratings were compiled to indicate the strength and 
weakness of each solution. 

The culmination of completing a performance analysis is the recommendation of solutions. A variety 
of solution recommendations and alternatives were provided due to an assortment of causes for each 
organizational gap. As noted in Figure 5, the team developed blended solutions. Blended solutions are often 
more effective because they can be re-purposed to develop collateral materials that will assist the performer 
when transferring or apply solutions in the workplace, and short-term as well as long-term solutions can be 
developed (Schaffer & Douglas, 2002). 

Final presentation 

During the final presentation, the project team reviewed the whole process of the project as a re- 
alignment strategy. The gap and cause analysis report was presented with the recommendations that related 
systematically with the potential solutions. Next, the potential solutions were presented with a description of 
how the solutions were selected, rated, and blended in order to meet the business needs. Furthermore, a list of 
actions, including communications, training, and work environment, were provided to the client as the answers 
of “What next?’’ or “What does it all mean?’’ Those actions were presented as the crucial steps and processes 
that should be considered by the client. Open discussion followed for the client to select the potential solutions. 

An additional constraint surfaced during this meeting as it became evident that the client organization 
did not want to exercise too much control over the target performers. The client representatives made it clear 
that they were not willing to take actions such as revoking the user’s computer access, even if the performer was 
not acting according to information security best practices. 

The outputs of this performance analysis included the identified performance needs, environmental 
barriers, causes and needs for improvement. These outputs provided the client representatives with a glimpse 
into attitudes and actions of their target performers. In addition the causes that were identified could later be 
used to facilitate the evaluation of solution effectiveness, since reaction, learning and skill transfer evaluation, 
as well as cost-benefit evaluations could be related to these causes. 

Overall, the client representatives were pleased with the solutions presented, as evidenced by their 
asking of the project team for assistance in designing, developing, and implementing some of the proposed 
solutions. 



Discussion 



Challenges and Constraints 

The project had several challenges and constraints. The target performer population was too broad and 
had diverse characteristics. The population size was about 10,500 with a variation of backgrounds in terms of 
descriptors such as major areas of study and class level. Additionally, due to the time constraint, the project 
team had to shorten the data collection to one week with a focused group of subjects that may not have been a 
representative sample of the population. Therefore, the project team considered the questionnaire as a pilot 
study that could be continued and improved upon in the future. Nevertheless, the data collected allowed the 
client and the team to interpret the data with effective tools, and to identify individuals’ current performance as 
a baseline. At the same time, the client was able to learn more about the data-driven systematic process. 

As Robinson and Robinson (1995) mentioned. Block (1981) has identified three consulting styles that 
are used with frequency: “the pair-of-hands, expert, and collaborative styles’’ (p. 18-22). Only the collaborative 
style will yield results for a performance consultant. The team experienced “the pair-of-hands’’ throughout the 
process as they continually attempted to keep the client informed about, and involved in, the performance 
analysis process. In addition, on some occasions, some of the processes were modified to meet the client’s 
needs. For example, the project team could not prioritize the major causes of gaps during the data review 
meeting. The cause prioritization was conducted by emailing the client a form to fill out. Though the client 
completed the form, the rationale provided was below the project team’s expectations. Similarly, due to the time 
constraint, the project team had to alter the original plan for the select phase which should have included two 
meetings with the client in order to brainstorm, present and rate the potential solutions with the input of the 
client. Instead, the project team, independent of the client, rated the solutions, and then presented them to the 
client during the final meeting. The feedback of the client was later retrieved via email from the course 
instructor. 
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Lessons Learned 

Collaboration is the Key to Success The most important lesson that the team learned while conducting 
the performance analysis is that getting the client involved in all phases of the analysis process greatly improves 
the process. This can be done by keeping the client informed of the project status via e-mail, status memos, 
minutes, and agendas. Again, as mentioned by Robinson and Robinson (1995), Block (1981) indicated that it 
will bring 1 + 1 = 3 (synergy) when the client and the consultant work collaboratively. In addition, keeping the 
client representatives involved and sharing HPT visual models and frameworks with them helped to increase 
their understanding of the performance analysis process. This performance analysis experience was positive for 
all involved because of the involvement of various stakeholders throughout the entire process. Some of the 
feedback from the client is as follows: 

“I would give both groups all 5's across the board for Organization, Professionalism, Quality, Results, 
and Outcomes. The only thing I wish we could have had more time and thought put into was the 
survey process. But given the constraints, I think this was well done also.” 

“The computer security team presented us with some useful suggestions and I am planning on 
implementing (or at least trying to implement) some of their suggestions. I even spoke with our 
Communications AVP about the give-aways and she thought that was possible.” 

In October 2004, the project team contacted the client for comments about the effectiveness of the 
proposed solutions. Some comments are as follows: 

“We are in the process of getting the online class up and running. We are working with a vendor to 
obtain prizes as suggested by your group to entice students to participate in our course.” 

“I think there is some improvement. We have had an increase in the number of downloads of our free 
Anti-virus software from our web site.” 

Visual Models are Effective The performance analysis approach was introduced to the client during 
the process. The use of visual models, such as the Performance Relationship Map, Performance Pyramid, and 
system model proved to be effective in helping the client understand the processes. For example, the use of the 
Performance Pyramid aided the project team in helping the client to look to solutions beyond training to 
solutions that addressed gaps other than those caused by a lack of knowledge and skills. The client 
representatives enjoyed the use of visual models, as evidenced by the following feedback they provided when 
asked if the project provided any opportunity for them to learn about the HPT approach: 

“Yes. I think the most interesting concepts were breaking down the lack of performance into the 
different categories, such as knowledge & skills verses desire, expectations and the other categories 
defining why someone is performing or not.” 

“I think the visual models were effective in showing the different reasons for performance gaps. I 
found that breaking the questions into categories that specifically addressed each of the areas of the 
pyramid was very helpful in understanding reasons for the performance gaps.” 

Conclusions 

During completion of the project, the team gained specific knowledge about the performance analysis 
process. They learned that it is difficult, but possible to create solutions for an organization that has little 
control, or that chooses to exercise little control over the target performer. In addition, the team learned that it is 
often necessary to remind the client of the data to make sure that the process remains data-driven. Furthermore, 
the team became more aware of information security issues (the context of the project). Finally, the project 
team learned that visual models could be effective tools for educating clients about, and involving them in HPT 
processes. 

In conclusion, the significance of the project is that it provided a real world context in which the 
project team and the client could learn about HPT processes. The novices on the project team learned how to 
apply HPT processes and work effectively with a client, and the client representatives learned how to assess and 
address performance gaps with varied solutions. In addition, the experiences were enriched and the consultant- 
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client relationship continued by the request from the client to continue the HPT process after the analysis 
project through the design, development, implementation, and evaluation of the solutions. 
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